Closes #595 (the VRAM half of it — this adds the automatic trigger, not the manual load/unload UI button).

Problem

On a headless/server deployment there's no way for models to unload automatically. They load on demand at generate time, but the only things that ever unload them are an explicit POST /models/unload or process shutdown (app.py). There's no idle path.

So an instance that served one request an hour ago is still pinning its weights — ~5.6GB for qwen-tts-1.7B, which is most of an 8GB card. Anything else sharing that GPU is starved until you restart the container or remember to call the endpoint by hand.

Change

An asyncio reaper that unloads models after a period of inactivity.

• Off by default. Gated on VOICEBOX_IDLE_UNLOAD_SECONDS; 0/unset disables it entirely, so existing behaviour is unchanged for anyone who doesn't opt in.
• Reuses the existing unload primitives — the same tts / transcribe / llm unload calls already used on shutdown, with per-model failure isolation so one failure can't block the others.
• Can't race in-flight work. "Idle" is read from the serial generation queue's own state (_queued_generation_ids / _running_generation_tasks), not inferred externally, so a model is never pulled out from under a queued or running generation. A long generation continually re-arms the countdown.
• The next request reloads the model transparently — it just pays the cold-start it already pays on first use.

Three files: a new backend/services/idle_unload.py, two lines in app.py to start it, and a mark_activity() call in the generation worker's finally block (the one place every generation exits through, success/failure/cancel alike).

Testing

Manual, on an RTX 2060 Super (8GB), Docker/CUDA, qwen-tts-1.7B:

Reclaims ~4.3GB of resident model weights. Verified separately that:

• the reaper does not fire during a running generation (watched it hold off across a ~200s job that peaked at 6504 MiB, with a 120s idle timeout that would otherwise have been eligible mid-job);
• the model reloads correctly on the next request after an auto-unload, and that generation completes (status=completed, no errors);
• with the env var unset, the reaper never starts.

I did not run Black across the touched files — app.py has pre-existing formatting drift (the all_origins CORS block) and reformatting it would add unrelated churn to this diff. The new file and my task_queue.py edit are both Black-clean on their own.

Disclosure: this patch was written with AI assistance (Claude), as was this description. It's tested end-to-end on my own build and the numbers above are real measurements from my hardware, not generated. If AI-assisted contributions aren't welcome here I understand — happy for this to be closed, and it stays a local patch on my build.

Summary by CodeRabbit

• New Features
  • Added optional automatic unloading of speech and language models after a configurable idle period.
  • Unloading is coordinated with generation activity to avoid interrupting queued or in-progress work.
  • Idle activity tracking now resets after each generation attempt, including failures and cancellations.